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§0. Introduction 

Orbits of the coadjoint action play an important role in harmonic analysis, theory 
of dynamic systems, noncommutative geometry. A. A. Kirillov's orbit method 
allows to reduce the classification problem of unitary representations of nilpotent 
Lie groups to the classification of coadjoint orbits [U, E]. This makes possible 
to solve the problems of representation theory in geometric terms of the orbit 
space. But it turns out that the classification of coadjoint orbits is a difficult 
problem itself. In particular the classification problem of coadjoint orbits of the 
unitriangular group UT(n, K) for an arbitrary n is far from its solution pD, B| 
H]. The classification of regular orbits (i.e, orbits of maximum dimension) was 
achieved in the pioneering paper on the orbit method [2]. The classification of 
coadjoint orbits for n < 6 is known pD, §6.3.3]. 

Number of coadjoint orbits of given dimension over the finite field ¥ q is po- 
lynomial in q. In [31 S] a correspondence between these polynomials and Euler- 
Bernoulli (/-polynomials is conjectured; this conjecture was checked by computer 
for n < 11. Authors of the paper [8] made computer program finding coadjoint 
orbits over the finite field (for their investigations in completely integrable Toda 
lattices). 

Every coadjoint orbit in the space ut*(n, K) is contained in some orbit of left- 
right action of UT(n, K) x UT(n, K). In [3, El E] the classification of orbits of 
left-right action is presented and correspondent representations are studied. 

In this paper we consider the case n < 7 and subregular orbits of an arbitrary 
dimension. The paper consists of three sections. The first section contains the 
classification of coadjoint orbits of the unitriangular group for n < 7 over an 
arbitrary field K of zero characteristic (Theorem 1.9). The description of orbits is 
given in terms of admissible subsets; it's convenient to represent them as diagrams. 
In the next Theorem 1.10 we solve the classification problem of coadjoint orbits 
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for n < 7 in terms of canonical forms. For any canonical form the polarization is 
constructed. This allows to classify unitary irreducible representations of the real 
unitriangular groups and absolutely maximal primitive ideals of the respective 
universal enveloping algebras (Theorem 1.12 and Corollaries). Note that this 
diagram method doesn't work for the unitriangular group of an arbitrary size. 
Authors have constructed counterexamples to Theorems 1.3 and 1.10 for n = 9. 
At the end of the first section we prove that the number of orbits of Borel subgroup 
T(n, K) in ut*(n, K) is infinite if n > 5 (Theorem 1.15). 

As usual, we identify the symmetric algebra S(q) with the algebra K[q*} of 
regular functions on 0*. Coadjoint orbits of the group UT(n, K) are closed, 
because orbits of a regular action of a nilpotent group on an affine variety are 
closed (pH, 11.2.4]). Let be the ideal of functions that equal zero on the orbit 
H in S(q). This ideal is an absolutely maximal ideal (AMP-ideal) with respect to 
the natural Poisson bracket in S(g). The map Q i— ► establishes the bijection 
between the set of coadjoint orbits and the set of AMP-ideals. In §2 we describe 
the set of generators of an arbitrary AMP-ideal for n < 7 (Theorem 2.2). This 
allows to represent an orbit as a set of solutions of polynomial equations. It turns 
out that one can present the system of generators as a system of some polynomials 
of the form P — c, where P is a coefficient of a minor of the characteristic matrix 
$(r), c G K . Authors believe that the last proposition (see Corollary 2.4) is true 
for all coadjoint orbits of the unitriangular group of an arbitrary dimension. 

The description of regular orbits is given in terms of minors of the matrix $ ( [2] 
and Theorem 3.1). In §3 of the paper (Theorem 3.3) we present the classification 
of subregular orbits (i.e., orbits of dimension dim£l reg . — 2). The solution is given 
in terms of the coefficients of minors of the characteristic matrix. 

§1. Admissible diagrams and canonical forms of orbits 

Let K be a field of zero characteristic. The unitriangular group UT(n, K) consists 
of the matrices 

/ 1 ... \ 
a 2 i 1 ... 

\ a n ,2 ■■■ 1 J 

where a^- G K. The Lie algebra g := ut(n, K) of the group G := UT(n, K) 
consists of lower-triangular matrices with zeroes on the diagonal. Let A + := A+ 
be the system of positive roots of gl(n, K). All positive roots are oiji(h) = Xj — Xi, 
where j < i and h = diag(xi, . . . , x n ). 
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For arbitrary pair i > j denote by yij the matrix from g with 1 in the 
th entry and zeros elsewhere. The matrix y^ is an eigenvector for ad^ with 
eigenvalue —oiji- We say that the pair corresponds to the positive root otji. 
For a positive root £ = otji we also denote by the matrix yij. 

Matrices {y^, i > j} form a basis of the Lie algebra g. By <£> we denote the 
following formal matrix: 

/ ... \ 
$= yn o ... 

\ynl Vn2 ■■■ o J 

Let a > (3 be the lexicographic order on A + . I.e., if pairs and (s,t) 

correspond to the roots a and (3 resp., then a > (3 means that j < t or j = t, 
i > s. In other words, a > [3 if the entry of the n x n-matrix lies at the left 
of (s,t) or in the same column, but lower, than (s,t). 

There exists the Killing form (x, y) = Tr xy on the Lie algebra £jt(n, K). The 
Killing form is non-degenerate over a field of zero characteristic. This allows to 
identify g* with the subspace of upper-triangular matrices with zeroes on the 
main diagonal. We also identify symmetric algebra S(g) with algebra if [fl*] of 
polynomials on g*. There exists the natural Poisson bracket on S(g) defined by 
the formula {x, y} = [x, y] for all x,y G g. 
Definition 1.1. 

1. An ideal I in S(g) is a Poisson ideal if {/, 5(g)} C /. 

2. We say that an ideal I is an absolutely maximal Poisson ideal (AMP-ideal) if 
the ideal I <S>k K' is a maximal Poisson ideal in S(g <S>k K 1 ) for any extension 
K' / K of the ground field. 

One can see that an ideal I is Poisson if and only if it's G-invariant. The map 
/ i— > Ann / establishes the bijection between the set of all AMP-ideals of S(g) 
and the set of coadjoint orbits in g*. 
Definition 1.2. 

1. We say that A C A + is an additive subset if a + (3 G A whenever a, (3 G A 
and a + (3 G A + . 

2. Let M C A. If a G A, (3 G M and a + (3 G A+ imply a + /3 G M, then we say 
that M is a normal subset of A. 

If A is an additive subset of A + and M is a normal subset of A, then gA '■= 
span{y 7 : 7 G A} is a subalgebra and m := span{y 7 : 7 G M} is an ideal of qa- 
For an arbitrary ^ in an additive subset A C A + consider 

C^A) = { 1 eA: £> A7 }. 
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The subset C(£, A) splits into the union C(£, A) = C+(£, A) UC_(£, A), where 
C+(£,A) (resp. C_(£, A)) consists of 7 G C(£,A) such that 7 > £ — 7 (resp. 

7 < £ - t)- 

Denote by A(£) the subset A \ C(£, A). One can see that A(£) ia also an 
additive subset. 

Definition 1.3. The subset 5 = {£1 > . . . > £&} C A + is called admissible if it's 
constructed by the following rule. The first element £1 is an arbitrary element of 
A\ := A + . Let A2 := Al(£i). The second element £2 is an arbitrary element of 
A2. The next element £3 is an arbitrary element of A3 := ^2 (£2) etc. 

Let A := := A k+1 and M := M(S) := A \ S. Note that M is a normal 

subset of A. 

One can split the subset S into the union of two subsets S®USn- By definition, 
& G S®, if A 7^ A+i (i-e., C(&, Ai) 7^ 0), and £ 4 G S D , if A l = A i+1 (i.e., 

C(^,A) = 0). 

To each admissible subset S = {£1 > . . . > £&} we assign the diagram D(S) 
which is said to be admissible. The diagram D(S) is the n x n-matrix. Its 
entries are empty for all i < j; if i > j, then (i, j)-th entry is filled by one of the 
symbols 0, □, •, ±. In order to construct the admissible diagram D(S), consider 
the empty n x n-matrix D. Let (ii, Ji), i\ > Ji, corresponds to the first root £1 
in 5. If i\ — ji > 1 (resp. z'i — ji = 1), then put the symbol <S> (resp. the symbol 
□) into the (ii, Ji)-th entry of Put the symbol • into the entries (a, b),b < ji, 
and (a, ji), a > i\. Put also the symbol + into all entries (a, ji), 1 < a < z'i, 
lying in the ji-th column, and the symbol — into all entries (z'i,a), ji < a < z'i, 
lying in the z'l-th row. 

Suppose that the entry (z'2, J2), ^2 > J2, corresponds to the next positive root 
£2 in S. Put the symbol • into all empty entries (a, 6), 6 < j'2, and (a, J2), a > *2- 
If there are empty pairs of entries (22, a) and (a, j'2), where ]2 < a < ^2, in the 
"lower-triangle" part of our matrix (i.e., among the entries with i > j), then 
put the symbol ® into the entry (12^2) and the symbol — (resp. +) into the 
entry (22, a) (resp. (a,^)). If there are no empty pairs of entries in the "lower- 
triangular" part of considering matrix, then fill the entry (22, J2) by the symbol □. 
Note that if one of the entries (22, a), (a, J2) is already filled on the previous step, 
then we don't fit the other. Then we apply this procedure to the roots £3, . . . , £& 
in S. If, after all, there are an empty entries in the "lower-triangular" part, then 
put the symbol • into them. As result, we have the admissible diagram D(S). 

Note that entries of D(S), that corresponds to the roots in M(S), S®, Sb, are 
filled by the symbols •, (g), □ respectively. Recall that A(S) = M(S) U^U Sn- 
Example. To the admissible subset S = {«i3, «25, «35, ^34} of the system of 
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positive roots for n = 5 we assign the diagram 
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The construction of the diagram splits into four steps: 
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Here M(S) = {0:15,0:14} and = {015,014} U S. Note that the subsets 

{013,025,035}, {013,025,034} and {013,025} are also admissible. Corresponding 
diagrams are 
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These diagrams have common S^-part and different Sb-parts. 
One can see that if the subsets S = S® U Sa and S' = S® U S' n are admissible 
then the subset S® U (S'n U S' a ) is also admissible. 

Definition 1.4. An admissible subset 5 is called maximal if its subset S a is 
maximal. 

It's convenient to construct maximal admissible subsets for the unitriangular 
Lie algebra of given dimension by the following "sequence rule". The first maximal 
admissible subset is the so-called regular subset S^eg. Its first root £1 is the 
maximal root of A\ := A + , the next root £2 is the maximal root in A2 := ^4i(£i) 
etc. (see the diagrams (n,0,l) for n = 3,4,5,6,7 above). Suppose that the 
maximal admissible subset S = {£1 > . . . > £ p > . . . > £&}, where £ p is the 
minimal root in S®, is already constructed. Then the next maximal admissible 
subset S' = {£[ > . . . > £' > . . . > £' m } is constructed as follows. The roots ^ 
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coinside with the roots £j for 1 < i < p — 1, the root is the maximal root of 
{ij G A p : £ p > 77}, and the root ^ is the maximal root of Ai for all i > p. 

This procedure allows to list all maximal admissible diagrams. Here we list all 
maximal admissible diagrams for n = 3, 4, 5. Diagrams for n = 6, 7 are listed at 
the end of the paper (we omit empty cells in those diagrams). The number of the 
diagram consists of three numbers (n, k, m), where n is the dimension of matrix, 
/c is the number of symbols • in the first column and m is the serial number of 
the diagram in the set (n,k). Every set (n,k) ends by the symbolic diagram 
(n, &,*), which corresponds to the set of the diagrams: one can delete the first 
row and the first column and replace the (n — 1) x (n — l)-block by an arbitrary 
(n — 1) x (n — l)-diagram. 
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(3, 0, 1) 
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(4, 2, *) 



(5, 0, 1) 



(5, 2, 1) 
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(5, 0, 2) 



(5, 2, 2) 
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(5, 2, 3) 
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(5, 3, *) 



Definition 1.5. Consider the set <S n , which consists of all pairs (5, c), where 
S is a maximal admissible subset of A + := A+, and c is a map S — > X such that 
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c (0 7^ for £ G S®. Any pair (5, c) is called maximal admissible, and the set S n 
is called the set of maximal admissible pairs. 

Let the extension c: A(S) — > X of c be given by 0(7) = for 7 G M(S). 
Definition 1.6. Let (5, c) be a maximal admissible pair. We say that G 0* 
is a canonical form if the following condition hold: 

r x / 0(7), ecjin 7 G 
/5 ' cW ~ \ 0, ecjin 7 GA+\A(5). 

Note that the restriction of to £U(S) is a character of 
The unitriangular Lie algebras form the sequence 

g = ut(n, K) D ut(n - 1, iT) D ... D ut(2, X), 

where ut(i, X) consists of matrices from ut(n, X) with zeroes in the first n — t 
columns. Systems of positive roots of the Lie algebras {gl(t, K)}2<t< n also form 
the sequence A+ D A+_ : D . . . D Aj. 

Denote A^ = and A^ = A+_ t+1 \ A+_ t . One can see that 7 G A^, if y 1 
lies in the £-th column of the matrix <£>. The system of positive roots splits into 
the union 

A + = A (1) U A (2) U . . . U A (n ~ 1} . 

We also put A® := A^nA, M® := A^fW S® := A^nS, 4° : = A^nS®, 

Sg } := AWnfib. 

Let m(l) = 1. We denote 

m(f) = max{j | ^ G U . . . U + 1, 

B t = ^n-t+l n An(t)- 

Note that A^ = B t fl A^. Subsets are additive, they form the sequence 

A+ = B 1 D5 2 D...D £ n _i D 5 n = 0. 

To each admissible subset S of A + we assign the sequence of the diagrams 
{D t (S)}i< t <n constructed as follows. The diagram Di(S) (of dimension n x 
n) coincides with D(S). One can construct the diagram D t (S) of dimension 
(n — t + 1) x (n — t + 1) by the following rule: 

1) delete the first t — 1 rows and t — 1 columns, 

2) omit the symbols (more precisely, the symbol { — }) corresponding to the 
weights, which don't contain in B t . 

Example. D(S) = (5,2,1). 
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The subsets Bi, B 2 , B i} B4 consist of the roots corresponding to all non-empty 
entries of the diagrams D\, D 2} D% } resp. 

Denote by g t the subalgebra of ut(n — t + 1, K) generated by {y 1 : 7 G B t }. 
These subalgebras form the sequence 

= 0i D . . . D Q t D Qt+i D . . . D Q n = {0}. 

Let I t be the ideal of S(Q t ) generated by y 1 — 0(7), 7 G A^. It's easy to check 
that I t is a Poisson ideal of S(Q t )- 

Denote by A m the Poisson algebra K\pi, . . . ,p m ; qi, . . . , g m ], where {pi, = 1 
and {pi, qj} = for i ^ j. Let Aq = X. 

A Poisson algebra C is called a tensor product of a Poisson algebras B\ and B2 
\iC = B 1 ®B 2 and {B 1 ,B 2 } = 0. 

Lemma 1.7. Let n < 7. For any £ there exists the Poisson inclusion 

t : S( flt+ i) - S( flt )/J t (1.1) 

such that the Poisson algebra S(gt)/h splits into the tensor product of two Poisson 
subalgebras 

S(to)/It = A®e t S{B t+ i), (1.2) 

where the Poisson subalgebra A is isomorphic to Ak t for a certain k t . 
Proof. Using the list of maximal admissible diagrams for n < 7, we conclude 
that there are only three cases of filling of a ^-column: 1) the t-th column contains 
one symbol <S> and the symbols □ are situated below then the symbol <S> (or don't 
appear in this column); 2) the i-th column contains one symbol <S> and one of 
the symbols □ is situated above then ®; 3) the t-th column doesn't contain the 
symbol <S>. 

Case 1. The t-th column contains one symbol ® and the symbols □ are situated 
below then the symbol (g) (or don't appear in this column). Then S$ consists of 
the one root; denote this root by £. 

Consider the subsets C + := C+(£, B t ) = {71 > . . . > 7^} and C := 
C-(£,B t ) = {71 < . . • < 7^}, where £ = 7; + 7- for 1 < % < m. The sub- 
set A® is the union 

A {t) = {£} U M (<) U S$ U C+. 
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The subset B t+1 has the form B t+1 = B t \ (A® U CI). 

Take the maximal element 71 G C + and the minimal element 7^ G C_ (recall 

that 7l + 7 ; = £)- 

Let := \ {7[} . The subset B' t is additive. Indeed, if j[ = 77 + 77', where 
77, 77' G A, 77 > 7/', then 71 + 77 G C+. This contradicts the choice of 71. 

By g' t denote the subalgebra generated by y v , 77 G i^. The elements p = 
2/7(3 Q = UjiU^ 1 satisfy the condition {p,q} = 1 and generate the subalgebra 
isomorphic to A\. 

We claim that {q, a} — mod I t for an arbitrary a G S(g' t ). Indeed, for an 
arbitrary 77 G B' t the root 71 + 77 is contained in Since 77 7^ 7^ we conclude 
that 71 + 77 7^ ^. Since 71 is a maximal element of C + , we have 71 + 77 ^ C + . A 
root £j from S\j can't be equal to a sum of two roots from the correspondent A4, 
so 71 + 77 ^ 5b. Hence, 71 + 77 G and so [y 7l , t/J = mod 1^ for an arbitrary 
77 G We conclude that {q, a} = mod 1^ for all a G S(g' t ). 

By definition, put 




s=0 



where a G <S'(^) and &d p (a) = {73, a} . 

It's easy to check directly that {73, a} = {q,a} = mod I t and {a, 6} = 

{a, 6} mod ([ID], 4. 7. 5). The map aha can be extended to the morphism 
of Poisson algebras & : S(Q l t ) — > S(Qt)/h- Denote by 1[ the ideal of S(Qt) gener- 
ated by I t and y lx . The ideal I[ coincides with the kernel of 9' . Extend 6' to the 
monomorphism 

0' : S(d)/I' t - 

The Poisson algebra S(Q t )/It splits into the tensor product Ai <S> lui6'. 

Then, for an arbitrary 1 < m < k we consider the additive subset B^ = 
B t \ {7^, . . .j' m }. By q[ denote the subalgebra generated by y v , 77 G B^ m \ and 
by l[ denote the ideal generated by l[ m ^ and y lm . As in the case m = 1, one 
can construct the monomorphism 

«<"> : 5(gj m) )/4 m) - 

The Poisson algebra S(g t )/It splits into the tensor product A m <8> Im#( m ). For 
m = k we have 

S( f } )/4 fc) = ^(flm) and S(g t )/I t = A k ®Im6 {k) . 
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Case 2. There are only two diagrams D t (S) such than one of their t-th columns 
contains one symbol <S> and one of the symbols □ is situated above then <g>: 











□ 








+ 








<8> 









L> 4 (7,3,4) 
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04(7,3, 



For these diagrams we construct the required inclusion directly. Here q is 
the Lie subalgebra generated by y v corresponding to the non-empty entries of 
diagrams. 

1) D±(7, 3, 4). The ideal I is generated by the elements y±i — C\, 7/21 — £2, where 
c\ 7^ 0. The factor algebra S(q)/I coincides with the algebra A\ generated by 

P = V 43V 41 and 9 = 2/31- 

2) D±(7, 3, 8). The ideal I is generated by the elements 2/41 — C\, 1731—02 with C\ 7^ 0. 
The factor algebra S{q)/I has the form A\ <g) ^[^32], where ^32 = 2/32 - c^VxiVzx 
and *4i is generated by = y^Uu and q = y<i\- 

Case 3. The t-th column doesn't contain the symbol <g>. Then A® = U 
The natural inclusion Qt+i ~^ Qt can De extended to the isomorphism S(Qt+i) ~^ 
S(9t)/It, □ 

Let Q t , I t} Pi,qi be as in Lemma 1.7. Denote 

X t :=AnnJ t Cfi£, 

X tfi := {/Gl t : Pi(f) = qiU) = flJifl 1 < i < A*}. 

Let 6^* be the map X t — > induced by the inclusion # f (see (1.1)). Since 1^ is 
a Poisson ideal, then X t is a Poisson submanifold and 6 t * is a Poisson map. 
Corollary 1.8. We claim that 

1) the map to \— > ^^ 1 (w) is a bijection between the set of all symplectic leaves of 
Q* t+l and the set of all symplectic leaves of X t ; 

2) the restriction 

pt ■= 0t*\x tfi 

is a bijection between X t $ and Qt + \', this restriction coincides with the restriction 
of the natural projection Q t — » to X t $. 



10 



Proof. An arbitrary Poisson ideal of the tensor product A k ® B has the form 
Ak <S> J ', where J is a certain Poisson ideal of B. So, the first part of the corollary 
follows from (1.2). 

To prove the second one, we note that for an arbitrary a G S(gt+i) and / G X t $ 
we have a(f) = a(f). Indeed, it follows from (1.3), that 

To a maximal admissible pair (S,c) we assign the AMP-ideal Is,c of S(g), 
which is constructed by induction on the number of column 1 < t < n — 1: 

C 1^ C . . . C 1^ =: l s , c . 

By definition, the ideal coincides with I\. According to Lemma 1.7, 
^(g)//! = A kl <8> 6>i(5(fl 2 ))- Suppose that the ideal X w such that 

S(9)/lV=A k ®e(S(Q t+1 )), (1.4) 

where = &i + . . . + ^ and = 9\ . . . 6 t} is already constructed. Then the ideal 
j;(t+i) contains and is generated modulo by 6(I t +i). 
So, 

S(9)/T (t+1) = A k ® 6(S(g t+1 )/l^) = A k ®6 (A kt+1 ® 6 t+1 (S( Qt+1 ))) 

= A k+kt+1 ®ee t+1 (s(Q t +2)). 

Denote by 2# )C the ideal T^ n ~ l \ Since g n = 0, then 

S( Q )/l s , c = A kl+ ... +kn _ 1 . (1.5) 

Let Qs )C = Ann2s c . It follows from (1.5) that the ideal Is,c is a AMP-ideal, 
so ls,c coincides with the defining ideal X(Qs,c) of the orbit (this ideal consists 
of all elements of S(g) such that their restrictions to the orbit Q$,c are equal to 
zero). 

Theorem 1.9. Let n < 7 and g = ut(n, K). 

1) The map (5, c) 1— > 2s )C is a bijection between the set S n of all maximal admis- 
sible pairs and the set of all AMP-ideals of S(g). 

2) The map (5, c) 1— > Qs,c is a bijection between the set S n of all maximal admis- 
sible pairs and the set of all coadjoint orbits of the group UT(n, K). 

3) dim(Qj) is equal to the number of symbols ± in the diagram D(S). 
Proof. Part 3) follows from (1.5). Part 2) is a corollary of 1). Let's prove the 
first part of the theorem. 
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Let X be an AMP-ideal of S(g). If X contains all elements y n i, . . . , 2/21 of the 
first column of the matrix <£>, then denote by X^ the ideal (y n i, . . . , 2/21) and 
consider the second column. 

If X contains y n i, . . . , and doesn't contain yn, then the ideal X^ gener- 
ated by (y n i, . . . , is a prime Poisson ideal of 5(g). The element yn is a 
Casimir element modulo X^\ Hence, the ideal X contains a certain element of 
the form yn — c, c G K . Consequently, X contains the ideal X^ generated by X^ 
and yn — c. Denote by £1 the positive root that corresponds to the entry (i, 1). 

According to Lemma 1.7, S{q)/X^ = Ak® 0\S{q2)- Consider the elements 

%n2),- ■■,0l(2/32) 

(they correspond to the basic vectors from the second column of the matrix $). 
In the same way as for the first column, we can construct S = {£1, . . .} and 
c : S — > K, starting from the ideal X. □ 

Theorem 1.10. Let n < 7 and let (S 1 , c) be a maximal admissible pair. The 
canonical linear form f$, c is contained in the orbit Qs,c- 

Proof. Let {g t } be the sequence of subalgebras corresponding to S as above. 
Corollary 1.8(1) allows to construct the set of coadjoint orbits (or, equivalently, 
symplectic leaves) Q t £ Qt (°f the coadjoint action of the groups Exp^) resp.) 
such that for an arbitrary t the orbit Q t is contained in X t and Q t = 6^{VL t+ i). 

It clearly follows from Corollary 1.8(2) that the orbit Qs,c contains the element 
Pi 1 o • • • o ^"^(O); this element coincides with fs, c - 1=1 

Corollary 1.11. The ideal X$ c is generated by the elements Q v — Q v (fs,c) 
with 77 G A(S) and Q v {fs,c) = c(r]). Here an arbitrary element Q v has the 
form + R >r} , where R >r) is a certain element of the subalgebra generated by 
{y~,, 7 > V}, and R >v (fs,c) = 0. 

Proof. The ideal is generated modulo X^ 1 ^ by the elements O t -i(y v ) — c{rj), 
77 G A^. It follows from (1.3) that there exists the element Q v = y v + i? >r? of 
5(0) (where R >v is as above) that coincides with Ot-^y^). Finally, 

Q v (fs,c) = (y v + R> v )(fs, c ) = y v {fs lC ) = 0(77). □ 

Let Q be a Lie algebra and f E G*. A subalgebra p is called a polarization of /, 
if p is a maximal isotropic subspace with respect to the skew-symmetric bilinear 
form /([#, y]) on Q. 

Theorem 1.12. Let n < 7 and S be a maximal admissible subset. Let ps be the 
linear subspace spanned by the basic vectors {y v } such that the root 77 corresponds 
to the symbols +, (g), □ or • in the diagram D(S) (i.e., not to the symbol — ). If 
S doesn't correspond to the diagram (7,3,8), then ps is a polarization of all linear 
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forms fs >c - To construct the polarization p (7,3,8), ° ne should replace 7/54 by 7/75. 
Proof is by direct enumeration of all diagrams of dimension < 7. □ 

The classification of coadjoint orbits and construction of polarizations allow to 
classify all unitary irreducible representations [T] and all primitive ideals of the 
universal enveloping algebra |10j . 

Corollary 1.13. An arbitrary unitary irreducible representation of the group 
UT(n, R) with n < 7 is induced from a certain one-dimensional representation 
e ? / = fs,c, of the subgroup Ps = exp(ps). The maximal admissible pair 
(5, c) is uniquely determined by the representation. □ 

Corollary 1.14. An arbitrary absolutely primitive ideal of U(ut(n, K)) with 
n < 7 is induced from a certain ideal of U(ps) Ker f\p s , where / = fs, c - The 
maximal admissible pair (£, c) is uniquely determined by the ideal. □ 
Let T(n, K) be the Borel subgroup of GL(n, K). The adjoint action of the group 
T(n, K) maps the Lie algebra of unitriangular matrices q := ut(n, K) to itself. 
This defines the action of the group T(n, if) on g*. 

Theorem 1.15. The number of T(n, if) -orbits in 0* is finite for n < 5 and 
infinite for n > 5. 

Proof. The group T(n, i^) maps a given orbit to a certain orbit H^c', which 
corresponds to the same maximal admissible subset S and the other admissible 
map d . 

One can see that for n < 5 an arbitrary set Qs of coadjoint orbits {&s,c} (with 
fixed S and arbitrary admissible map c: 5 — > ET) is a T(n, i^)-orbit. So, for 
n < 5 the number of T(n, i^)-orbits in 0* if finite. 

Then, consider the diagram (6,3,4) for n = 6: 















+ 
























• 


+ 


□ 








• 




□ 








• 


• 


• 


□ 


□ 





This diagram corresponds to the maximal admissible subset S (it consists of 
six elements). Codimension of Qs,c m is equal to six. But codimension of 
arbitrary coadjoint UT(n, i^)-orbit in the correspondent T(n, i^)-orbit is less or 
equal to five. Hence, the number of T(n, i^)-orbits is infinite. 

For arbitrary n one can construct an example of infinite set of T(n, i^)-orbits, 
using the diagram of the following form: its last six rows and six columns form the 
diagram (6,3,4) and all other entries of this diagram are filled by the symbol •. 
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Note that an orbit from the set (6,3,4) has the following defining equations 
(the the next section): 

2/61 = ?/51 = 2/41 = 2/62 = 2/63 = 0, 2/31 = Ci 7^ 0, Z/ 52 = c 2 7^ 0, 



2/42 2/43 
2/52 2/53 



C3, 2/532/31 + 2/522/21 = C 4 , 2/64 = C 5 , 2/65 = C 6 - □ 



§2. Absolutely maximal Poisson ideals 

In this section we'll present the system of defining equations of arbitrary coadjoint 
orbit for n < 7. Consider the so-called characteristic matrix for the formal matrix 
$ (see §T 



$( r ) = r$ + £" = 



/ 1 

T2/21 





1 



0\ 




\ tj/„i rz/ n2 • • • 1 / 

Recall that we identify S(g) with the algebra of polynomials on g* (we also 
identify g* with the space of all upper-triangular matrices with zeroes on the main 
diagonal). 

Every minor of the (formal) characteristic matrix 4>(r) is a polynomial on r 
with the coefficients in the symmetric algebra S(g) = if [fl*]. Values of these coef- 
ficients on an element F G g* coincides with the coefficients of the correspondent 
minor of the usual characteristic matrix tF + E. 

According to Theorem 1.9, to an arbitrary admissible pair (£, c) one can assign 
the AMP-ideal Is,c and the orbit Qs,c = Ann2s )C . Dimension of the correspondent 
orbit is equal to the number of the symbols ± in the diagram. The ideal Is,c 
coincides with Z(Qs jC ). To each root 77 G A(S) one can assign the generator of 
the ideal Ts, c . In terms of the diagram D(S), to each symbol □, (g), • one can 
assign the generator of Is,c- In this section we'll show that there are coefficients 
of minors of the characteristic matrix $(r), which generate the ideal Ts,c- 

Let = {Pi > . . . > Pk®}, where k® = \S®\. Denote r] = £j — Si G A(S). 
The (i,j)-th entry of the diagram D(S) (i > j) is filled by one of the symbols □, 
®, •. To the root r\ assign the permutation 

w v := Sfc • • • Sf3 t s v , where Pi >-••>&> V > Pt+i > •-•> Pk®- 

Consider the set of the columns A := Aj = {1, . . . , j} and the set of the rows 

w v (A) = ord{^(l), . . . , w v (j)}. 



14 



The minor M^ A (r) of the matrix is polynomial on r: 

<aW = V + -V- </<...<<*, 

where degree g (resp. rf) of the least (resp. the leading) term is equal to 

q= |A\«yY| = KA\A|, 

d = |t{l < m < j | i m > m}. 
Let 0i, ... , be fundamental weights. 

Lemma 2.1. Let n < 7. We claim that for an arbitrary rj = Ej — Ei € A(S) the 

weight (1 — w v )(f)j can be uniquely represented as a sum 77 + ^ /3, where j3 runs 

over a certain subset H(S, rj) C {/3i, . . . , f3 t } C 5®. 

Proof is by direct enumeration of all diagrams for n < 7. □ 

Denote /i = h(S,r}) := 77) | + 1. Note that h is uniquely determined by 77 

and S. 

Theorem 2.2. Let n < 7. An AMP-ideal ls,c of 5(g) is generated by the ele- 
mentsP M -P0 7? G^,r fl eP0 r? = P M (/ 5 , c ). 

Proof. One can prove this theorem for every maximal admissible subset S. For 
example, consider the subset (7,2,7): 

















+ 














+ 


+ 












+ 





























• 


• 


+ 


□ 


□ 






• 


• 




□ 


□ 







The additive subset A(S) has the from A(S) = U Sq U M(S), where 

#® = {«15, «24, «37}, = {«46, «47, «56, «57}, M(S') = {c*i 6 , CK17, a 2 6, «27}- 

Let X be the ideal generated by the elements P/j )7? — P^ with 77 G 74(5). To 
prove that X = 2s )C , it's enough to check that 

1) X is a Poisson ideal, 

2) J(/ 5>c ) = 0, 

3) the ideal X is generated by the elements y v + T >T] — c, where 77 G ^4(5) and T >r} 
is contained in the subalgebra generated by {t/ 7 , 7 > 77}. 
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Indeed, 1), 2) imply X C ls,c- By 3) the algebra S(q)/X is isomorphic to the 
algebra of polynomials K[y v : 77 £ A + \ A(S)}. Therefore the ideal X is prime 
and DimZ = |A + \ A(S) | = dimf^. We conclude X = Xs iC - 

Let us show that X really satisfies 1), 2) and 3). Denote P^ = Ph,rj, where 
77 = otji. By definition, the ideal X is generated by P^ — Ffj where Ffj = Pij(fs,c) 
and runs over pairs which are filled by symbols ®, •, □ in the diagram. 

We consider the lexicographical order on the set on monomials of yij such that 
Va. > Vj] if ol < V- Here we list the polynomials P^- starting from the leading term: 

P71 = 2/7i, Pei = 2/6i, P51 = 2/5i, P51 = 2/51, P72 = 



2/51 2/52 
2/71 2/72 



P62 — 


2/51 


2/52 


, Pt2 — 


2/41 


2/42 




2/61 


2/62 




2/51 


2/52 



^73 — 



2/41 2/42 2/43 
2/51 2/52 2/53 
2/71 2/72 2/73 



P74 — 2/74 



P64 — 



2/41 2/42 
2/51 2/52 

2/41 2/42 
2/51 2/52 



+ 2/73 



2/31 2/32 
2/51 2/52 



= 2/74P42 + 



2/63 2/64 
2/73 2/74 



= -2/642/73^42 + 



^75 = 2/752/51 + 2/742/41 + 2/732/31 = 2/752/51 + 



^65 — 



2/63 
2/73 



2/65 
2/75 



2/51 + 



2/41 = -2/652/732/51 + 



2/63 2/64 
2/73 2/74 

The ideal X contains the Poisson ideal X^ generated by the elements 2/71, 7/6 1, 
2/51 — 2/51- The elements P62, P72 are Casimir elements modulo X^K Extend the 
ideal X^ to the ideal J := {X^\ P 72 , P 62 ). 

One can directly check that the above polynomials P42, P73, P74, P64, P75, P65 
are Casimir elements in the factor algebra modulo J . Therefore the ideal X is a 
Poisson ideal; this proves 1). The proof of 2) is obvious. 

Choose the new system of generators in X that consists of 2/71, 2/61, 2/5 1 
2/73 - 2/73, Qij - Qij where (i,j) ranges over (7, 4), (6, 4), (7, 5), (6, 5) and where 
Qij is defined similar to P^- changing 2/51, 2/73, A2 by 2/51, 2/73, ^42- This proves 3). 
□ 

Corollary 2.3. An orbit ri^ )C has the defining equations P/^ = const (77), where 
77 ranges over positive roots that correspond to the symbols •, <g> and □ in the 
diagram D(S). □ 

Corollary 2.4. A coadjoint orbit for n < 7 has defining equations of the form 
P — c, where P is a certain coefficient of a minor of the characteristic matrix and 

ceK.u 



2/5V 
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§3. Subregular orbits 



In this section we'll describe all subregular orbits of the unitriangular group (The- 
orem 3.3). Denote no = [§] , n® = [^y-] , A" = n ^ n ~ 1 ^ , Note that n = Uq +7^ + 1. 

At first, we'll recall the description of regular orbits. One can see that the 
minors 

Pj ■■= K-Ai^ l<j<no (3.1), 
(top indices are numbers of columns, bottom indices are numbers of rows) of the 
matrix $ are Casimir elements of S(g). 

Theorem 3.1 [2] • The defining ideal Q reg of a regular orbit Q reg is generated by 
the elements P± — ci, . . . , P no — c no , where Cj G K (and cj ^ for 1 < j < n®), 

We'll recall a proof of this theorem and show that regular orbits correspond to 
regular maximal admissible subsets (see §1). Note that I'SVegl = no and l/S^eg,®! = 
n®. 

Now we define the polynomials Zi, . . . , Z n(gi , which are needed for the sequel. 
Consider the set of the minors 

Pj(r) ■■= M^f +1 _ n (r), l<j<n-l, 

of the characteristic matrix $(r). For 1 < j < no the polynomial Pj(r) is equal 
to PjT 3 , where Pj is the correspondent minor of 

For j > no the least term of the polynomial Pj(r) equals PjT n ~K Denote 
by Z n _j the coefficient of r n ~ J+1 in the polynomial Pj(r), j > no- Note that 
1 < n — j < n — no = n® + 1, because Uq < j < n — 1. In particular Z\ = 
Vn,n-\Vn-\,\ + • • • + 2^,22/2,1 is the coefficient of r 2 in the polynomial M 2 , -^~ (r). 

The following Proposition 3.2 is needed for the sequel. Consider the decompo- 
sition of the space X := q* to the subsets X = Xq U X\ U . . . U X n _i, where 

Xi = {/ G X\ f(y nl ) = ... = f(y n -i +ltl ) = 0, f(y n _ hl ) + 0}. 

Denote d{ = max{dim^| Q C Xi}. We say that an orbit is z-regular, if Q C Xi 
and dim Q = di. 

A maximal admissible subset S = {£i > . . . > is called i-regular (denote 
it by S(Xi)) if £i = ai >n -i and an arbitrary element 2 < £ < k, is maximal 
element of the correspondent A f (see Definition 1.3). Note that entries of the 
diagram D(Xi) corresponding to S(Xi), which are filled by the symbols •, are 
exactly (n, 1), . . . , (n — i + 1, 1). 
Proposition 3.2. We have 

^ _ f N - (no + 2i), if i < n®, 
\ A" - (no + 2rc®), if i > n®. 
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Proof. To each S(Xi) we'll assign the dense in Xj set of orbits {^sfXAc} °f 
equal dimension. On the other hand, the set of orbits of maximum dimension is 
open pj], §2.6]. Hence, d{ = dim fW^Ac- 

This set of orbits {^sprAc} w iH De constructed differently in different cases 1) 
and 2). 

Case 1. i < n®. See diagrams D(X2) for n = 7 and n = 8. 



□ 



□ 



□ 



□ 



□ 



Columns of D(Xj) satisfy conditions from the cases 1) or 3) from the proof of 
Lemma 1.7. This allows to construct the ideal Ts(Xi),c Dv methods of Theorem 
1.9. The dimension of the orbit £ls(XA,c 1S equal to the number of the symbols 
± in the diagrams. All orbits from this set have equal dimension, which equals 
N — = N — (no + 2i). This concludes the proof in the fist case. 

Note that one can find the ideal X s ^ Xi ),c using methods of §2. Indeed, let 
{£i > . . . > £ no } be the first no roots of S(Xi) (i.e. the weights corresponding to 
the symbols <S> in the odd case and the weights corresponding to the symbols <g> 
with the weight a nQ ^ no+ i in the even case). 

To each weight 1 < j < no assign the row number m(j) such that ^ = oei )m u). 
Consider the system of the minors Q±, . . . , Q nQ , where 

Q j = M 1 >7tf rv 

We also consider the ideal J generated by y n i, . . . , y n -i+iA- It's easy to check 
that the polynomials Qx, . . . ,Q no , Z\, . . . , Z{ are Casimir elements modulo J. 
The ideal Ts{Xi),c i s generated by J and the elements of the form P — c, where 
P = Qj, 1 < 3< n , or P = Z h 1<j<i. 
Case 2. i > n®. See diagrams D(X±) for n = 7 and n = 8: 
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□ 



<g> 
□ 



® 
□ 
□ 



Columns of I}(Xi) also satisfy conditions from the cases 1) or 3) from the proof 
of Lemma 1.7, so we can construct the ideal T S ( Xi ),c by methods of Theorem 1.9. 
The dimension of ft>s{Xi),c equals the number of the symbols ± in the diagrams. 
All orbits from this set have equal dimension, which is equal to N — \S(Xi)\ = 
N — (no + 2?!®). This concludes the proof in the second case. 

Note that one can find the generators of Zs(Xi),c as m the first case: the ideal 
Ts(x t ),c is generated by J and the elements of the form P — c, where P = Qj, 
1 < j < + 1, or P = Zj, 1 < j < n - i - 2. □ 

Remark. We've just shown that an orbit Qs(Xi),c 1S ^-regular. But if i > 1, then 
an ^-regular orbit doesn't have the form Qs(x t ),c m general. 

Proof of Theorem 3.1. We can apply methods of Theorem 1.9 to regular 
diagrams (i.e., diagrams D(S Teg )); i.e., we can construct the set of orbits {^s reg ,c} 
of dimension N — no, starting from S ieg . Since this set is dense in g*, we conclude 
that the maximum dimension of an orbit is equal to N — no. The ideal X(S Teg ) is 
generated by Pj — Cj, 1 < j < no- 

So, it's enough to check that an arbitrary regular orbit coincides with one of 
the orbits constructed by a regular diagram. Let Q be a regular orbit. It follows 
from Proposition 3.2 that Q C Xq. Since y n \ are Casimir elements of S(q), that 
T(Q) contains a certain element y n \ — c with c ^ 0. Let = {y n \ — c). Using 
Lemma 1.7, we obtain 

S(q)/1 {1) = A n -2 ® 5(ut(n - 2, K)). 

There exists the symplectic leaf (orbit) uj in ut*(n — 2, K), corresponding to the 
symplectic leaf (orbit) in q*. Since the orbit Q is regular, then the orbit to is 
regular in ut*(n — 2, K). To conclude the proof, it remains to apply induction on 
n. □ 

Let's now describe subregular orbits. For n = 3 and n = 4 they are described 
by the diagrams: 
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□ 






• 


□ 























+ 










□ 








® 










• 


+ 






• 


□ 


□ 






• 


® 






(4, 1, 


1) 


(4, 2, 


1) 



(3, 1, 1) 



One can check that for n = 5, 6, 7 diagrams corresponding to subregular orbits 
are: (5,0,2), (5,1,1) for n = 5; (6,0,2), (6,0,3), (6,1,1) for n = 6; (7,0,2), (7,0,3), 
(7,1,1) for n = 7. 

For an arbitrary 1 < j < consider the minors 

p' — M 1 '-^ P" — A/fivJ-lJ+l 

r j - lvI n-j,n-j+2,...,n r j ~ IVI n-j+l,...,n 

of the matrix which border the minor Pj-\ from (3.1). For even n consider 



also the minor PL 

n 



ii^-l,2,. ..,n -l 

n ,n +3,no+4,...,n - 



P' -c 



P" 

Jo 



Recall that the restriction of a minor Pj (from (3.1)) to an arbitrary orbit is 
constant, because these minors are Casimir elements. 
Theorem 3.3. Let f2 sreg be a subregular orbit. 

1) If P n0 (Q sveg ) ^ 0, then there are the number 1 < jo < n® and the numbers 

' ~?o — 1' ^j^j Cj _|_2? • • • i Cn — 1 

}ciT, {c no ,c}cK 
(with c no 7^ for odd n) such that the ideal I(Q sreg ) is generated by the elements 
P, - a, where i = 1, . . . , j - 1, j + 2, . . . , n , 

2) If P n0 (Q sveg ) = and n is odd, then there are the numbers 

such that the ideal 7(i7 sre g) is generated by the elements 

Pi - Cf, where z = 1, . . . , n® - 1, 

3) If P n0 (Q sveg ) = and n is even, then there are the numbers 

{ci,...,c n0 _i,c'} C X*, {c",c} C K 
such that the ideal I(fl sreg ) is generated by the elements 



Pi — Ci, where i = 1. 



- 1. 
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and the elements of the form a) or b) , where 



Proof. The proof for n < 4 is by direct calculations. Let f2 sreg . be a subregular 
orbit in ut(n, i^) for n > 4. It follows from Proposition 3.2 that there are two 
cases for n > 4: Q srcg C Xq or Q srcg C Xi. 
Cases 1. n sreg C Xi. 

The ideal X(Q sreg ) contains the ideal J generated by y n \. The elements P^, 
3 < z < no, and y n -i,i, Un2, Z\ are Casimir elements modulo J . There are 
{c, c' l5 c", C3 . . . , c„ } C X such that the restrictions to f2 sreg of all functions from 
the ideal X generated by J , Pi — ci, 3 < % < no and y n -i,i — c', y n 2 — c", Z\ — c 
with c' 7^ 0, are equal to zero. Hence, X C Z(n sreg ). 

Consider the Poisson ideal 2q = (y n ,i,yn-i,i ~ d,Z\ — c). Easy to see that 
lode Z(£l srcg ). According to Lemma 1.7, 

S(q)/1 = An-ztoSfa), 

where the Lie algebra 02 is isomorphic to ut(n — 2, X). Both of ideals X, I(£l sreg ) 
are generated modulo Xq by a regular AMP-ideal of £(02)- Thus, X = X(Q sreg ). 
Case 2. £l srcg c Xq. 

Suppose that Pi(Q STeg ) ^ 0, . . . , P jo -i(Q sieg ) ^ and P jo (Q sieg ) = 0. The 
proof is similar to the first case (one should replace y n i, y n -i 1, y n 2 by P/ , Pj , 
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List of maximal admissible diagrams for n = 6 









.0,1) 




+ 








4 


□ 




4 




— 










— 




-1 


* 




m 


,M) 




■ 

+ 






4 


4 


□ 






— 


— 






« 








□ 


* 




/ft 

m 


2,1) 


















* 


+ 


□ 


J-l 


* 








•¥ 




ft 


,2, 


5) 


4 


□ 














• 




□ 4 




# 


• 


• 




-1 


4- 




(6, 3,3) 










* 


+ 


4 








4 






• 


• 












it 


m 










« 


□ 


□ 






• 


* 


4 




■ 


• 


* 




-1 



+ 


+ 






Z) 


+i 










4 


* 


□ 


□ 




® 


— 






-1 


■ 

+ 


■ 

4 


(6,1,2) 






— 










— 


— 




# 


* 


□ 


o 


□ 




+ 


(6, 


% 


2) 












* 






4 




* 


* 


□ 


© 


-1 


4 


□ 


(6, 




6) 












• 


* 


o 


□ 




* 


• 


* 


* 


□ 


+ 




(6,3,4) 


* 


4 


□ 






* 




□ 






* 


• 


* 


□ 


□ 


+ 

© 








8) 


* 


□ 


□ 








• 


* 


□ 




* 


• 


• 


• 


□1 



4 

4 


□ 




,0,3) 


4 


• 






4 


• 








% 


— 


- 


- 




4 
4 


LI 




3) 


4 


• 


4- 




® 










* 


• 


® 




□ 


4 

4 


+ 


ft 












* 






□ 




* 


* 


□ 


* 




4 






3,1) 










* 


4 


4- 






+ 


$ 






• 




D 




-1 


4 




(W) 










* 


□ 


4- 




* 


* 


4- 


□ 






* 








□ 




(6. 


4,*) 










* 




* 




* 












* 


* 




* 



4- 
+ 


o 


(6,0, 


4) 


+ 


• 


□ 






+ 


• 


* 


□ 




© 


— 






-1 


4* 

♦ 

+ 


U 


(6,1,4) 


4 


• 


□ 
















* 


* 


* 


□ 


□ 


4- 

4* 


□ 


(6, 




4) 


® 










• 




+ 


□ 




* 


* 




□ 


-1 






(6,3, 


2) 


* 


+ 


a 






* 




« 


□ 




* 




□ 




-1 


* 

+ 




(M,6) 




□ 


+ 








* 








• 


* 


* 


□ 


□ 
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List of maximal admissible diagrams for n = 7 
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(7,0,5) 
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+ 
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-1 
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* 
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J. 
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□ 
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■ * 
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4- 
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+ 


+ 
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4 


• 
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□1 
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L,6) 


4- 
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• 


□ 
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* 


® 


— 


□1 
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(7, 1, 10) 
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